Towards local visual modeling for image captioning

In this paper, we study the local visual modeling with grid features for image captioning, which is critical generating accurate and detailed captions. To achieve target, propose a Locality-Sensitive Transformer Network (LSTNet) two novel designs, namely Attention (LSA) Fusion (LSF). LSA deployed intra-layer interaction in via relationship between each its neighbors. It reduces difficulty of object recognition during captioning. LSF used inter-layer information fusion, aggregates different encoder layers cross-layer semantical complementarity. With these proposed LSTNet can model to improve captioning quality. validate LSTNet, conduct extensive experiments on competitive MS-COCO benchmark. The experimental results show that not only capable modeling, but also outperforms bunch state-of-the-art models offline online testings, i.e., 134.8 CIDEr 136.3 CIDEr, respectively. Besides, generalization verified Flickr8k Flickr30k datasets. source code available GitHub: https://www.github.com/xmu-xiaoma666/LSTNet.